execution, db: bind StateCache fills to transaction views and reject stale fills - #22444

Closes #22356 — the forward stale-fill direction: a fill from a read view older than the latest apply is rejected. The reverse direction around unwind is tracked in #22463 and stays open (details in the scope section).

What StateCache is, who touches it, when

StateCache is one process-global in-memory cache of the latest committed values of accounts, storage and code. It exists to skip the file-accessor/MDBX stack on repeated GetLatest reads. It is not a snapshot: per key it holds one value — the newest committed value known to the process, whether applied by execution or filled from a read. It is on by default; USE_STATE_CACHE=false constructs no cache at all (readers go straight to the backing tx), and STATE_CACHE_FILLS=false disables every reader write (apply-only mode) — the kill switch and the A/B lever for measuring what fills contribute.

A fill is a cache write performed on behalf of a database reader (fill-on-miss); an apply is the authoritative post-commit write from SharedDomains.Commit. A read view is what a read-only temporal tx sees (its MDBX read tx + pinned files view); its frontier is the exclusive txNum end of what it can see — a view with frontier N sees txNums < N. This sharing — including the RPC fills — is main's code today (wiring: node/eth/backend.go hands execmodule.Cache to the embedded rpcdaemon as its kvcache; CacheView.Get → SharedDomains.AsGetter → getLatestMetered, which consults and fills the shared cache). The PR does not add the sharing; it adds the missing ordering.

Problem

1. Reader begins a read-only tx; its frontier is N.
2. Exec at txNum M >= N deletes key K and commits; the post-commit apply empties K's slot.
3. Reader reads K: cache miss → reads its own older tx → gets the pre-delete value → its fill re-inserts it (the slot is empty, so PutIfAbsent lands).
4. Anyone's next GetLatest(K) serves the deleted value from the cache.

Each step is consistent inside its own tx. The bug is the unordered hand-off between two txs through the shared cache — no per-tx consistency property can prevent it.

sequenceDiagram
    participant Reader as Reader with frontier N
    participant Cache as shared StateCache
    participant Exec as Exec commit
    Exec->>Cache: apply at txNum M >= N. delete K, appliedEnd = M+1
    Reader->>Cache: GetLatest(K). miss, the slot is empty
    Reader->>Reader: falls back to its own older tx. pre-delete value
    Reader->>Cache: fill(K, pre-delete value)
    Note over Cache: main. PutIfAbsent lands, K is resurrected
    Note over Cache: this PR. rejected, frontier N < appliedEnd

Solution

1. API: no reads or writes on the cache object, only handles

StateCache has no data methods (like db). Access goes through two handles (like tx):

• ReadView — reads plus admission-gated fills, bound to one tx's read view; it must not outlive the tx. SharedDomains.AsGetter*(tx) builds one per getter; the plain GetLatest wrappers use a frontier-less view and bind a frontier only on the miss path. The view resolves its frontier internally, so a fill can no longer be paired with another tx's frontier.
• Applier — Apply / Unwind / Clear. Used by the authoritative SharedDomains commit/unwind paths (every SD holds a handle; the operations serialize under the admission lock).

2. Fill admission

The cache tracks, per domain, the exclusive end of what has been applied (appliedEnd), guarded by one RWMutex:

• Applier.Apply (write lock) advances appliedEnd and mutates the cache in the same critical section. Applying txNum X records appliedEnd = X+1.
• ReadView.Fill (read lock) rechecks its view's frontier and inserts before releasing: admitted iff frontier >= appliedEnd. Fills are put-if-absent, so they never replace a live authoritative entry.

Ordering proof, two cases: if the fill takes the lock first, the later apply overwrites or deletes it; if the apply takes the lock first, the stale fill is rejected. Equality is safe because a view whose frontier equals the applied end has seen the apply. Checking freshness only at view creation would not be enough — an apply can land between the check and the fill — which is why admission happens at fill time, under the same lock applies take.

3. Exact frontier

DomainVisibleEnd reports the exact exclusive frontier of a tx's domain read view: a file covering [0,N) reports N; a hot-DB view containing txNum N reports N+1. Views without an exact frontier answer ok=false and never fill — remote and history-disabled backends, and dependency-clamped values views (their reads mix ages, and the clamped-away state can appear later without any apply); such reads still work, they just skip cache population. Two memos, one per path: read-only temporal txs memoize the frontier per domain in a tx-local cache, reset only by ForceReopenUnderlyingFilesTx (which can only extend it); the writable path memoizes in SharedDomains, reset at flush and on ViewID change. A compile-time assert guards each memo's bitmask against domain-count growth.

Positive entries are stamped with their step-derived txNum bound. A negative entry is stamped with the last txNum its read view included, max(frontier-1, 0): since Unwind(N) treats N as the first rolled-back txNum, the negative survives Unwind(N) and is invalidated by Unwind(N-1).

Admission safety needs one more invariant: a view's frontier never decreases in a process that fills the cache (the DB component is frozen at tx begin; a files reopen only extends it). This is enforced, not just documented: wiring a fill-enabled StateCache forbids visibility lowering on that aggregator, and recalcVisibleFiles panics on the one transition that breaks admission — a cached state domain's visible end decreasing — whichever entry point causes it. Raising visibility (unaligning a lagging entity) stays allowed, and apply-only caches (STATE_CACHE_FILLS=false) skip the forbid: with no fills there is nothing for a lowered frontier to poison.

4. Physical deletion

Apply physically deletes account, storage and code entries; on the authoritative side absence is represented by absence, with no deletion markers (a marker can be evicted, re-opening the fill window). Reader-side negative entries do exist — they are ordinary admission-gated fills, stamped as below. An account update invalidates the derived address→codeHash mapping; an account deletion also removes the address's code binding; an account-only deletion does not advance the code-domain frontier (that would suppress valid code fills for unrelated contracts). On authoritative code apply, bytes are cloned before hashing so cached code and its codeHash cannot diverge. SeedAddrCodeHash accepts only view-sourced account records (a cache-sourced record can lag the latest apply and would carry pre-apply state past the gate); a read error does not seed the zero-hash sentinel.

What this PR deliberately does not do

• Reads are not snapshot-isolated: a ReadView hit can be newer than the view's tx. That is the same direction the Overlay already serves (embedded RPC at latest reads exec's newest state on purpose), and a single-version LRU cannot give stable per-view reads without becoming a second kvcache — node/shards already provides that model for the remote daemon. The cache's contract, in the forward direction, is monotonicity: content never regresses behind the applied frontier; unwinds invalidate by epoch and floor.
• The reverse direction around unwind — a view opened before an unwind refilling a value from the discarded fork — is tracked separately in execution/cache: stale-fill admission is one-sided — pre-unwind read views refill dead-fork values, and unwound keys bypass the flush cache-apply #22463.

Performance

• Cache hits are unchanged: no frontier query, no admission lock.
• The memoized frontier read is ~2.4 ns/op, 0 allocs (Apple M2 Max); the memo adds +64 B/op to a read-only temporal tx (BenchmarkBeginTemporalRo/WithBlockSnaps; absolute numbers drift with unrelated changes on main).
• Getter paths (exec workers, RPC) build their frontier-carrying ReadView once per getter; frontier-less views elsewhere (GetCode/GetCodeSize fast paths) are by-value and allocation-free, and a frontier is bound — one allocation — only on cold fill/seed paths. A cold negative fill through the plain GetLatest wrappers is ~0.3 µs/op, 48 B/op, 2 allocs: one allocation is the fill itself, one is binding the frontier on the miss path — both amortize against the backing read they follow (the no-cache baseline of the same read is ~0.1 µs).

Testing

Production-path reproductions through embedded RPC (execmodule.Cache.View over a published SharedDomains overlay and a real temporal DB): account, storage and code resurrection from an old RPC read view, and code-of-a-deleted-account refill (pinning both the SD-level paired deletion and the cache-level accounts-frontier check on code fills). Plus: admission at the exact frontier boundary and a 20,000-round concurrent apply/delete vs fill interleaving; physical deletion and derived code-mapping invalidation; account-only deletion not suppressing unrelated code fills; the address→codeHash mapping seeding only from view-sourced records; exact file/DB frontier calculation and memo re-derivation after files-view reopen and after flush; read-ahead freshness, no-exact-frontier backends never filling; negatives surviving Unwind(N) but not Unwind(N-1); the admission frontier surviving Clear; the STATE_CACHE_FILLS=false apply-only switch; USE_STATE_CACHE=false constructing no cache; node close releasing every cache-budget reservation (a full EngineApiTester lifecycle); and the visibility-lowering assert. All cross-package tests exercise the public ReadView/Applier API. The three resurrection tests use no API introduced by this PR: ported unchanged to main, all three fail with the resurrected values served — they pin the bug, not the refactor.

Verified with go test (and -race) on execution/cache, execution/exec, db/state/execctx, db/kv/temporal, full-tree build, benchmarks with -benchmem, and repeated clean make lint runs.

How to review

Suggested order:

1. execution/cache/cache.go (package doc) and execution/cache/view.go — the contract and the API: Frontier, ReadView, Applier. Small files; everything else implements them.
2. execution/cache/state_cache.go — admission internals: appliedEnd, fillIfFresh, apply.
3. db/state/execctx/domain_shared.go — wiring: getters hold a ReadView, commit/unwind hold the Applier; getLatestMetered shows the full read chain (mem → cache → backing tx → fill).
4. execution/exec/blocks_read_ahead.go — the read-ahead fill path.
5. db/state/execctx/statecache_rpc_integration_test.go — the resurrection reproduced end-to-end through the embedded-RPC path, and the fix pinned.